Challenges Inherent in the Design and 
Implementation of After-School 
Intervention Programs for Middle Grade 
Underachieving Readers 

Justin Velten and Kouider Mokhtari 


Abstract 

In this brief report, we share three challenges we encountered when designing and 
implementing an after school intervention program for an ethnically diverse group of middle 
grade underachieving readers. We also offer practical solutions to help guide middle school 
teams in anticipating and addressing potential problems when putting in place literacy 
interventions in after school settings. 
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I n the Fall of 2013, we teamed up with representatives of one local middle school to design, 
implement, and evaluate an after-school intervention program aimed at improving the academic 
vocabulary knowledge and the reading comprehension performance of an ethnically diverse 
group of 36 underachieving sixth graders. The target middle school was located in a socio¬ 
economically and ethnically diverse community in the Southwestern United States (pop. 105,000). 
At the time of the intervention, the school had an enrollment of approximately 600 students in 
grades six through eight with a 25:1 student to teacher ratio. Students represented various 
ethnicities, including White (4%), Black (20%), and Hispanic (76%). In addition, approximately 
93% of these students were eligible for a free or reduced price lunch and nearly 40% were 
designated as Limited English Proficient (LEP). 

The students were selected for participation in the after school literacy intervention program 
because they were underachieving in reading, writing, and academic subjects such as science and 
social studies as indicated by poor end of course grades (C- or below) in language arts, science, and 
social studies, as well as low performance (30th percentile or below) on the reading portion of the 
State of Texas Assessment of Academic Readiness (STAAR) test (TEA, 2015), the state’s high-stakes 
standardized test, which is administered annually to middle grade students. School records indicate 
that these students were among 45% of the students in grades six through eight who were placed in 
remedial reading classes. 

In light of the identified needs for the target students, we selected Word Generation, a program that 
is research-based and has a track record of documented effectiveness related to strengthening 
students’ academic vocabulary and reading comprehension skills. Word Generation was developed 
by a research team affiliated with the Strategic Education Research Partnership (SERP) at Harvard 
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University (Snow & Lawrence, 2011; Snow, Lawrence, & White, 2009). This program has proven 
quite successful in strengthening middle grade students’ academic vocabulary and improving 
reading comprehension performance as documented in various published research studies (Hwang, 
Lawrence, Mo, & Snow, 2014; Lawrence, 2012; Lawrence, Capostosto, Branum-Martin, White, & 
Snow, 2012; Lesaux, Kieffer, Faller, & Kelley, 2010; Snow & Lawrence, 2011). 

The Word Generation program consists of a carefully coordinated set of curriculum materials that 
support the teaching of academic vocabulary and reading comprehension across the disciplines of 
language arts, science, mathematics, and social studies. These materials include a series of weekly 
instructional units, which introduce students to academic vocabulary words embedded in passages 
exploring topics or issues of interest to adolescents (e.g., steroid use among athletes, censorship of 
books, popular music). The curriculum then provides opportunities for students to use the new 
words in classroom discussions, debate, and writing. The overall purpose of the program is to 
strengthen students’ academic vocabulary knowledge, which then helps them develop the 
background knowledge needed for effective reading comprehension and learning across the 
disciplines. 

We implemented the program during the 2013 - 2014 school year and reported our findings in 
Mokhtari & Velten (2015). For purposes of this manuscript, we provide only a summary of the 
findings, but devote the remainder of the manuscript to a discussion of the challenges we 
encountered in designing and implementing the program in this after school setting. We also share 
practical solutions using lessons learned from our own experience implementing the program with 
the goal of guiding other middle school teams in anticipating and eventually addressing potential 
problems likely to be encountered in designing and implementing after school programs. 

By and large, the implementation of the literacy intervention with our target students proved to be 
quite successful. In summary, we can report that after only 34 hours of small-group instruction 
using Word Generation, we compared the performance of intervention students (n=36), who 
voluntarily participated in the program, to a school-recruited comparison group of student peers 
(n=36), who participated in other after-school instructional programs of their choice. Our findings 
indicate that the vocabulary knowledge of our intervention students improved significantly when 
measured by scores obtained on a pre- and post academic vocabulary test for the whole program, 
available along with Word Generation’s instructional units (see http://wordgen.serpmedia.org). 
Performance on this measure simply tells us whether the academic vocabulary words taught were 
actually learned by students who completed the program. The premise here is that an increase in 
word knowledge mediates an improvement in reading comprehension, which is the core purpose of 
our study. More importantly, as we expected, we found that intervention students read more 
proficiently at the end of the intervention than they did at the beginning when measured by the 
Group Reading Assessment and Diagnostic Evaluation [GRADE] (Williams, 2001), a standardized, 
group-administered test of reading comprehension ability. We further found that intervention 
students received nearly identical post-test performance scores on the GRADE test at the end of the 
intervention when compared to students in the comparison group after adjusting for pre-test 
reading ability score differences. The findings indicate that implementing evidence-based programs 
such as Word Generation can help narrow the reading achievement gap among skilled and less 
skilled middle grade readers. 

It is important to note that while these program outcomes are encouraging, we faced significant 
challenges relating to the complexities of implementing the program in a real-world after school 
setting. Some of these challenges pertain to teacher and student participation, attendance, and 
scheduling, which we addressed in collaboration with members of the after school team throughout 
the design, implementation, and evaluation phases of the program. 
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Challenges and Solutions 

We encountered several challenges throughout the design, implementation, and evaluation of our 
after-school reading program, including, but not limited to, identifying students who were 
underachieving in reading and who were likely to benefit from the program, selecting and 
providing training for instructional staff to implement the program, finding adequate instructional 
space within the school premises wherein to implement the program, motivating reluctant students 
to participate in the program, and making arrangements to conduct pre and post assessments to 
enable us to determine program impact. For purposes of this brief report, we highlight three 
important challenges and provide possible solutions for addressing these challenges, focused 
primarily on teaching quality, dose or amount of instruction, and fidelity of program 
implementation. 

Challenge #1: Teaching Quality 

Consistent with prior research findings relative to the role of effective teaching in schools 
(Allington, 2011; Darling-Hammond, 1999), we wanted to ensure the selection of an effective 
vocabulary instruction program curriculum. However, we also wanted to be confident that those 
delivering instruction for our target students were adequately prepared to do so. Although we 
selected an effective program, we had to adapt it in terms of instructional time and number of days 
per week it was implemented so as to fit the school’s daily schedule of activities. 

For instance, instead of the recommended 15 minutes per day, five days a week, we implemented 
the program at a rate of 30 minutes per day, four days a week. Because our teachers consisted of 
paraprofessionals (i.e., college students, many of whom were novice teachers), we provided 
extensive training, mentoring, and supervision to enhance teaching effectiveness. Each of the six 
teachers initially received two half-days of intensive training focused on an in-depth review and 
discussion of the Word Generation curriculum program, along with unit and lesson design and 
implementation. 

To monitor progress and provide instructional support, we also held bi-weekly debriefing sessions 
with the teachers to review progress made, discuss and resolve issues and problems emerging from 
small-group instruction sessions, and provide assistance with issues pertaining to data collection 
and related program logistics. Finally, two members of the project team served as mentors for 
teachers by providing personalized consultation and instructional support. This mentoring role 
proved beneficial for teachers in preparing and delivering weekly lessons and addressing issues or 
problems encountered in small-group instruction sessions. 

A second challenge that threatened teaching quality was an overreliance on the part of our teachers 
on two or three instructional strategies they learned during training sessions. During our intensive 
training sessions, we provided teachers with a suite of resources, including twenty or so 
instructional strategies, such as the K(What I Know), W(What I Want to Know) and L(What I 
Learned) tool, graphic organizers, Think-Pair-Share, that are designed to strengthen academic 
vocabulary and reading comprehension skills. We also supplied them with various resources to 
support student vocabulary growth, content learning, and engagement. Finally, we provided them 
with a lesson framework we prepared ourselves as a sample for use during the first week-long unit 
of instruction. To organize instruction during the first week-long unit, we included in our sample 
lesson plan two strategies (KWL and a sample graphic organizer) to support the topic and goals of 
the week-long unit. To help ensure quality of teaching, we offered to review weekly lesson drafts 
and assist with the development of these lessons prior to implementing them with students. 
However, in reviewing drafts of lessons teachers developed, we discovered that most of the 
teachers relied rather heavily on the sample lessons templates we provided when they developed 
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their own lessons for subsequent units. In doing so, they had a tendency to rely on the use of the 
two strategies we provided on the sample lessons rather than selecting other strategies taught 
during training sessions, which might have made their teaching more useful and engaging for 
students. To address this constraint, we added a brief training session during which we offered 
extra guidance for developing, implementing, and evaluating lesson effectiveness. We also provided 
one-on-one mentoring and support for each teacher to help strengthen instruction and eventually 
improve students’ vocabulary growth, reading comprehension, and content learning. This process 
was helpful in that teachers became more reflective and thoughtful about developing and 
implementing their lessons. 

Challenge #2: Dose or Amount of Instruction 

When working with underachieving readers, providing them with good instruction using an 
evidence-based curriculum is necessary, but not sufficient, for improved learning. Equally 
important is the dose or amount of instruction each student receives. In order to achieve the 
desired reading improvement outcomes for our target students, we designed the program to help 
ensure that each student had an equal opportunity to receive an adequate amount of high quality 
instruction throughout the program. 

We used prior research guidelines to provide students with a sufficient amount of instruction (SO¬ 
TO hours) at a minimum to positively influence their academic vocabulary and reading 
comprehension performance. These guidelines vary dependent upon students’ grade level, 
language proficiency, literacy skills, and content learning needs. Clay (2005), for example, 
recommends that struggling early grade readers receive 12-20 weeks of intensive instruction (30 
minutes per day, five days a week) in one-on-one tutorial settings where they are provided 
personal attention by a trained literacy professional. 

In small learning groups, however, where a single instructor works with approximately five or six 
students, literacy experts (e.g., Fountas & Pinnell, 2010) recommend 14-18 weeks of intensive 
instruction for early grade struggling readers and 18-24 weeks for upper grade readers. Yet other 
experts (Allington, 2011) note that remediating reading may require as few as 30-40 hours of 
instruction and as many as 75-80 hours of instruction to get them on par with their proficient 
reading peers. Consistent with these research guidelines, we designed the program to include at 
least 17-18 weeks of intensive instruction in small group settings, at a rate of approximately 30 
minutes per day, four days per week. 

We ran into a major challenge with respect to our goal to provide a sufficient and equitable dose of 
instruction to all students participating in the academic vocabulary program. Even though we had a 
dedicated 45-minute time block for instruction, the actual time devoted to instruction (rougly 30 
minutes devoted to actual teaching) varied depending on several factors, including, but not limited 
to, class sessions starting on-time, students attending every class session four days a week, and 
teachers having to address occasional class disruptions. First, the school did not have sufficient 
classroom space for our six groups of students. Three of the groups met in the school library, which 
provided an ideal learning environment for the students and their teacher. One of the groups met in 
a computer lab on Mondays and Tuesdays and in another assigned classroom space on Wednesdays 
and Thursdays. Two of the groups met in the school cafeteria, which provided a rather poor 
learning environment for students and their teachers due to a noisy air-conditioning unit, 
disruptions from other students walking in and out of the cafeteria, and/or changes in after school 
staff meetings. Second, groups did not always start class on time because teachers had to make sure 
all six students were present (or not) before moving to their assigned classroom spaces. Oftentimes, 
teachers were left to rely on informal testimonies of classmates that a particular student was out 
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sick, had detention, or was on school grounds but running late. Finally, classes were occasionally 
cancelled, frequently with short notice from the school administration, to accommodate other 
scheduled events such as testing, participation in school-wide activities, and/or class cancellations 
due to weather conditions. 

During the first four weeks of the program, we discussed the above challenges with members of the 
school administration, who were attentive and responsive to our needs. With their assistance, we 
were able to make some improvements in class attendance, disruptions to instruction, and 
scheduling of special events, which eventually helped lessen the impact of some of these challenges 
of providing all students with an equitable dose of instruction. These small improvements were also 
helpful in designing future reading intervention programs within the school. 

Challenge #3: Fidelity of Program Implementation 

A program is only as good as its implementation. Fidelity of implementation occurs when teachers 
use and deliver the content of the curriculum in the way that it was designed to be used and 
delivered. When changes are made in the presentation of the curriculum, it is often unclear what 
the effects on the students will be. In our after school instructional setting, it was challenging for us 
to adhere strictly to the ways in which Word Generation was to be implemented, especially given 
the diversity of backgrounds and experiences of our students and teachers. 

However, we followed research guidelines relative to implementation fidelity for program 
designers and evaluators suggested by various researchers (Meyers & Brandt, 2015) that helped us 
implement the academic vocabulary program with the desired degree of implementation fidelity. 
For example, teachers followed a lesson template, which included key components of the lesson, 
including introducing academic vocabulary words using specific instructional strategies such as 
KWL [5 minutes], reading passages exploring a weekly topic, [10 minutes], and engaging students 
in discussing weekly topics using academic words learned [10 minutes]. 

Every week, teachers completed a fidelity of implementation form indicating whether they used 
parts of the lesson as-is, modified it to student needs, or did not use the lesson format at all. In 
addition, they provided reflections guided by open-ended questions discussing what worked, what 
did not work, and what they would do differently in future lessons. We used these two forms of 
feedback to determine the extent to which lessons were implemented as expected. 

For purposes of this program, our desired degree of fidelity was that teachers used lessons as 
indicated or modified them slightly to fit instructional needs at least 80% of the time. In addition, 
we consistently monitored student attendance and supervised the day to day program to help 
ensure that the instruction provided was implemented as envisioned. Finally, we debriefed with 
teachers on a regular basis to provide them with guidance and support, as well as encouragement 
needed to implement the program with fidelity. 

Perhaps one of the most significant challenges we encountered related to teacher attrition. All six of 
our teachers were college students pursuing degrees in various fields, including elementary 
education, business, and community nursing. Although every one of our six teachers were well 
trained and mentored to provide small-group instruction to their assigned students, and they were 
totally committed to the program, they occasionally had circumstances which prevented them from 
fulfilling their obligations to the program. For example, during the first few months of the project, 
we lost our graduate assistant who assisted us with the coordination and supervision of the 
program. Her role was crucial because she helped us oversee all components of the program, 
including data collection. Although the graduate student resignation did not impact program 
implementation in any significant way, it created a disruption and stalled the program somewhat 
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until we hired a replacement. In addition, because our program lasted an entire school year (fall 
and spring semesters), we had to replace two of the teachers who graduated in the fall of the same 
year. Finally, we also had to replace and train two of the teachers, one in the fall and one in the 
spring semester, who left for personal or work-related reasons. Although these disruptions are 
inevitable when working in schools, they can make a big difference with respect to whether a 
program succeeds or fails. Fortunately, we were able to attend to these disruptions to help ensure 
that our program had a chance of being implemented with a reasonable degree of fidelity. 

Concluding Thoughts 

An important lesson we learned throughout this project is that despite best efforts to incorporate 
effective practices for educating young adolescents, schools should expect to face challenges when 
designing and implementing supplemental interventions for underachieving middle grade readers. 
It is possible for school teams to anticipate and effectively address these challenges when teachers 
share common values, goals, and strategies for improving student learning; when they take 
collective responsibility for student achievement;,and when they are empowered through strong 
school leadership and support. Consistent with lessons learned from our experience with this after 
school project, we suggest that schools take into account the challenges and solutions described in 
this brief report, which center around the three important issues, namely ensuring that 
underachieving students are exposed to high quality instruction, that the instruction they receive is 
of sufficient intensity and duration to result in improved literacy achievement outcomes, and that 
instructional programs selected are implemented equitably and with fidelity. 
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